In recent years, there has been a growing interest in using captured human motion data as examples to simplify the process of programming or learning complex robot motions. See, e.g., A. Nakazawa, S. Nakaoka, K. Ikeuchi, and K. Yoko, “Imitating human dance motions through motion structure analysis”, Intl. Conference on Intelligent Robots and Systems (IROS), pages 2539-2544, Lausanne, Switzerland (2002). Captured human motion has been used to develop algorithms for ‘learning from demonstration’, a form of learning whereby a robot learns a task by watching the task being performed by a human. See, e.g., S. Schaal, “Learning from demonstration”, In M. C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, chapter 9, pages 1040-1046. MIT Press (1997). One goal of ‘learning from demonstration’ has been to replace the time-consuming manual programming of a robot by an automatic programming process, solely driven by showing the robot the task by an expert teacher. Captured human motion has also been used in computer animation to ‘retarget’ motion of one articulated figure to another figure with a similar structure. See, e.g., S. Tak, O. Song, and H. Ko, “Motion balance filtering”, Comput. Graph. Forum, (Eurograhics 2000), 19(3):437-446, (2000). See also, e.g., S. Tak and H. Ko, “A physically-based motion retargeting filter”, ACM Trans. on Graphics, 24(1):98-117 (2005).
The majority of existing approaches to control a robot from human demonstration use three steps. First, human motion is recorded using a marker based optical system, a magnetic system, or a wearable mechanical device resembling an exoskeleton. The recorded motion is then used as an input to a constrained non-linear optimization procedure to generate a re-designed joint level robot motion. See, e.g., A. Ude, C. G. Atkeson, and M. Riley, “Programming full-body movements for humanoid robots by observation”, Robotics and Autonomous Systems, 47:93-108 (2004). Constraints are also imposed to enforce the robot's kinematic, dynamic, and balance constraints. See, e.g., Y. Tamiya M. Inaba S. Kagami, F. Kanehiro and H. Inoue, “Autobalancer: An online dynamic balance compensation scheme for humanoid robots”, In Int. Workshop Alg. Found. Robot, (WAFR), Lausanne, Switzerland, (2000). Given sufficient computation time, a retargetted motion can be optimized off-line to satisfy a prescribed performance measure subject to the constraints. The retargetted joint motion is then used as the desired joint command input to the robot's independent joint controller. In the absence of external disturbances, the generated robot motion is kinematically and dynamically admissible by the robot and can be executed by the robot's control scheme during run-time.
Such a process has limitations. First, the complexity of sensing and instrumentation used to record human motion is time consuming and often limiting for use outside the laboratory. Moreover, the off-line re-targeting solution based on optimization methods is generally not suitable for online or reactive control of robots in dynamic scenes or in the presence of external disturbances. These limitations restrict the applicability of the approach to laboratory settings with static environments. An effective system should be interactive and therefore computationally fast, should work in static and dynamic environments, and should not rely on special instrumentation for the human demonstrator nor the environment.